[SPARK-48664][FOLLOWUP] Update 4.0.0-preview1 by Yikun · Pull Request #63 · apache/spark-docker

Yikun · 2024-06-21T01:48:49Z

What changes were proposed in this pull request?

Remove wget -nv -O KEYS https://downloads.apache.org/spark/KEYS and gpg --import KEYS;, it works but not meet the security concern from DOI.
add add-dockerfiles.sh 4.0.0-preview1 support to address [SPARK-48664] Add Apache Spark 4.0.0-preview1 Dockerfiles #61 (comment)
Fix versions.json: consider java17 as default version, so we can remove java17 tag.

Why are the changes needed?

update 4.0.0-preview1

Does this PR introduce any user-facing change?

new release

How was this patch tested?

./add-dockerfiles.sh 4.0.0-preview1 , no diff

Yikun · 2024-06-21T01:50:39Z

29.02 + gpg --batch --keyserver hkps://keys.openpgp.org --recv-key 4DC9676CEF9A83E98FCA02784D6620843CD87F5A
29.03 gpg: keybox '/tmp/tmp.NsK96iEJwQ/pubring.kbx' created
30.05 gpg: key 4D6620843CD87F5A: new key but contains no user ID - skipped
30.05 gpg: Total number processed: 1
30.05 gpg:           w/o user IDs: 1
30.05 + gpg --batch --verify spark.tgz.asc spark.tgz
32.10 gpg: Signature made Tue 28 May 2024 04:39:57 PM UTC
32.10 gpg:                using RSA key 4DC9676CEF9A83E98FCA02784D6620843CD87F5A
32.10 gpg:                issuer "wenchen@apache.org"
32.10 gpg: Can't check signature: No public key
------

cc @cloud-fan maybe need to add gpg key according #54 (comment)

cloud-fan · 2024-06-21T02:13:30Z

@Yikun Done, not sure how soon it will be available

cloud-fan · 2024-06-21T02:15:01Z

versions.json

-        "4.0.0-preview1-java17",
-        "python3-java17"
+        "4.0.0-preview1-python3",
+        "4.0.0-preview1"


shall we tag the "all-in-one" image with this spark version only name?

Or we think it's better to exclude R support by default?

I'm neutral on this. As I mentioned only had some short discussion on it.

cc @zhengruifeng @HyukjinKwon @gengliangwang any idea?

[1] https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o/edit?disco=AAAAf2TyFr0

if this was the first version, I might have preferred to have all-in-one to have 4.0.0-preview1
but it's been like that already so I think it's fine

Actually, 4.0.0-preview1 haven't been tagged in the published image (apache/spark) yet.

So, we can tag 4.0.0-preview1 on 4.0.0-preview1/scala2.13-java17-python3-r-ubuntu. that means the apache/spark:4.0.0-preview1/scala2.13-java17-python3-r-ubuntu will have a new alias tag 4.0.0-preview1.

cloud-fan · 2024-06-21T02:17:06Z

BTW, there is no place mentioning add-dockerfiles.sh, neither the spark release process doc, nor the README of this repo. Shall we mention it somewhere and ask release managers to use it?

Yikun · 2024-06-21T02:29:37Z

Done, not sure how soon it will be available

seems already work.

there is no place mentioning add-dockerfiles.sh, neither the spark release process doc, nor the README of this repo. Shall we mention it somewhere and ask release managers to use it?

Yes, we can! Now, only mentioned in spark-docker CONTRIBUTING.md

Yikun · 2024-06-21T08:19:45Z

there is no place mentioning add-dockerfiles.sh, neither the spark release process doc, nor the README of this repo. Shall we mention it somewhere and ask release managers to use it?

I added the doc, maybe you could give some review from release manager view. See 7bc0c58

cloud-fan · 2024-06-21T12:47:24Z

README.md

+
+### Step 2. Publish apache/spark Images.
+
+Click [Publish (Java 17 only)](https://github.com/apache/spark-docker/actions/workflows/publish-java17.yaml) (such as 4.x) or [Publish](https://github.com/apache/spark-docker/actions/workflows/publish.yml) (such as 3.x) to publish images.


there is also a Publish (Java 11 only), shall we mention it?

It seems you didn't notice the first line before "or"? java 17 already mentioned, :)

for 4.x, java 17; for 3.x, java 17 and java 11.

shouldn't mention https://github.com/apache/spark-docker/blob/master/.github/workflows/publish-java11.yml ? Or we should remove this file if it's no longer in use.

cloud-fan · 2024-06-21T12:49:16Z

versions.json

      "tags": [
-        "4.0.0-preview1-scala2.13-java17-python3-r-ubuntu"
+        "4.0.0-preview1-scala2.13-java17-python3-r-ubuntu",
+        "4.0.0-preview1"


If other Spark versions (3.x) exclude R support by default, I think 4.0 should follow it.

Yikun · 2024-06-24T01:05:41Z

Merged. @HyukjinKwon @cloud-fan Thanks!

Update 4.0.0-preview1

143b78d

Yikun changed the title ~~Update 4.0.0-preview1~~ [SPARK-48664][FOLLOWUP] Update 4.0.0-preview1 Jun 21, 2024

cloud-fan reviewed Jun 21, 2024

View reviewed changes

Yikun added 2 commits June 21, 2024 14:33

Tag all-in-one image with 4.0.0-preview1

c520027

Add Create a new version and About images in README.md

7bc0c58

Yikun marked this pull request as ready for review June 21, 2024 09:05

HyukjinKwon approved these changes Jun 21, 2024

View reviewed changes

cloud-fan reviewed Jun 21, 2024

View reviewed changes

Update versions.json tag

89007c7

HyukjinKwon approved these changes Jun 23, 2024

View reviewed changes

Yikun closed this in b1b1019 Jun 24, 2024

Yikun mentioned this pull request Sep 26, 2024

[SPARK-49813] Publish Apache Spark 3.5.3 to docker registry #72

Closed


		### Step 2. Publish apache/spark Images.

		Click [Publish (Java 17 only)](https://github.com/apache/spark-docker/actions/workflows/publish-java17.yaml) (such as 4.x) or [Publish](https://github.com/apache/spark-docker/actions/workflows/publish.yml) (such as 3.x) to publish images.

Comments

Conversation

Yikun commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Yikun commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Jun 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yikun Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jun 21, 2024

Uh oh!

Yikun commented Jun 21, 2024

Uh oh!

Yikun commented Jun 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yikun Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yikun commented Jun 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Yikun commented Jun 21, 2024 •

edited

Loading

Yikun commented Jun 21, 2024 •

edited

Loading

Yikun Jun 21, 2024 •

edited

Loading

Yikun Jun 21, 2024 •

edited

Loading